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Virtual Biopsy (Translucency) View Adjustments 

In order to facilitate differing modalities (e.g. CT, MR) and the needs of different doctors 
using the MPR and endoluminal views in conjunction with the virtual biopsy mode, as 
well as to allow for adjustments based on changes in findings of diagnosing colon 
cancers, an interface for adjusting the colors and thresholds for the virtual biopsy view is 
needed. With the interface, a user can adjust the imaging values (e.g. Hounsfeld units, 
raw intensity values) to which a particular color is applied in the virtual biopsy view; 
these values are known as thresholds. Users can also adjust the actually colors applied to 
different imaging values. The interface, where the user adjusts thresholds by moving 
sliders, and uses a color-picking dialog to select colors (colors of adjacent thresholds are 
linearly interpolated across a range of imaging values), is shown in the accompanying 
figure below. 
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Collaborative Virtual Examinations 

The following presents a framework for the collaborative viewing of 2D/3D medical 
data. The old way to perform collaborative viewing of a dataset (see Kaufman patent) 
was to send the data to all the participants and then compute locally the views of the 
exam. The new proposed way is to a server-based approach in which the dataset remains 
at one location and the views of the examination are sent to the participants in real time. 

To begin a session, the principal participant initiates a session on the server. The server 
loads the required data and creates a stream of visual information to which the principal 
participant immediately connects. At this point, the other participants may connect to the 
stream and observe the visual exam data. Only one participant may control the content of 
the stream at a time. If a passive participant wishes to take control, he requests ownership 
from the primary participant who then may grant control. In this way, only one 
participant is in control at a time. 

The stream may be composed of multiple sub-streams. For instance, there may be three 
different views of the exam subject from different directions. Some or all of these views 
may be updated or changed. The sub-streams are re-composed on the individual 
participants 5 computers. 

A farther advantage of this invention is that image generation can take place on a high- 
powered server and that the participants are not required to have expensive computers, 
but only a commodity network connection to the server. 
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Some additional ideas for Colon and Fusion 

Fuse and overlay secondary information on top of the filet view or an overview or 
an endoluminal view or a curved MPR view or a double-oblique MPR view or 
any other non-standard 2D or 3D projection. 

a. The fusion process is an alignment procedure that matches us features in 
the two data sets. 

b. The overlay of information is a selective blending of the secondary 
information over top of the original information either by a simple 
weighted average of the two color images or by a selective (data sensitive) 
combination of the images (e.g., the overlaid image is a color+opacity 
image). 

c. The secondary information can be data from another modality such as 
PET, SPECT, MRI or CT scan. It can also be information derived 
(computed from) either the primary modality data set or from a secondary 
data set. No alignment is necessary if the secondary information is simply 
computed or derived from the original data. 

d. The advantage of the overlay of secondary information is that 
confirmation of suspicious findings is automatic because the information 
is available directly at the position of suspicion. Furthermore, if 
suspicious regions are offered by the secondary information (as in PET or 
CAD), then the viewer is drawn to the suspicious regions by their 
heightened visibility. 

A method as defined in Claim 1, wherein the organ is a colon and the view is a 
filet view of the colon. The secondary information overlaid could be PET 
information (see below). 

a. A colon filet view looks like: 




Stripe 1 
Stripe 2 
Stripe 3 
Stripe 4 
Stripe 5 
Stripe 6 
Stripe 7 



It is a projection of the colon that stretches out the colon based on the 
centerline. This view is created using a cylindrical projection about the 
centerline. This means that parts of the colon that were once curved are 
now straightened out so this view introduces significant distortion at areas 
of high curvature. However, the advantage of this view is that you 
immediately get to see a large amount of the colon surface in a single 
image. Some polyps may be behind folds or stretched out to look like 
folds, while some folds may be squeezed to look like polyps. 



In order to help differentiate polyps, stool, and folds, it would be 
extremely helpful to overlay information from a secondary modality. For 
instance, Positron Emission Tomography (PET) scanners register the 
amount of chemical uptake of radioactive tracers that are injected into the 
patient. These move to the sites of increased metabolic activity and thus 
show up extremely concentrated at cancer sites. Although the information 
from PET is not yet very detailed (it has a relatively low spatial resolution 
compared to CT), it is extremely helpful when overlaid or embedded over 
CT or other data. In this sample, the breast tumor is 
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Similarly, the highlighted PET data could be overlaid on top of the filet 
view to indicated probable cancers. This overlay can be blended in and 
out with variable transparency. Data from modalities other than PET, such 
as SPECT or MRI, can also be overlaid and variable blended with the 
data, or laid out next to the CT data in alternating rows, for example. 

3 . In order to help differentiate polyps versus diverticula in the filet view or other 
2D/3D projection view, pseudocolor the depressed and elevated regions (innie vs. 
outtie) differently. 

a. The same technique of overlaying and juxtapositioning secondary 
information from another modality can be used to provide additional 
information or clues from derived or tertiary information. For instance, 
shape information such as curvature derived about the colon surface can 
be used help pseudocolor the surface. It is particularly difficult to tell the 
difference between a depressed diverticula and an elevated polyp in a 
static image such as the filet view. To help differentiate them, the shape 
can be computed and it can be determined at each such region to either 
color or highlight elevated regions and to color or de-enhance depressed 
regions. Another example is the overlaying of potential polyp locations 
over the filet view. A Computer Aided Diagnosis can be run on the colon 
surface and the results projected onto the filet image or any 2D or 3D 
image. 

4. The translucent display that colon has can be expanded to use the values of a 
second modality (e.g., PET) instead of just the CT data. This is helpful because 



5 



the PET data can be mis-registered by several mm and be hidden under the 
normal surface. 



a. In the colon system, there is a view called the translucent display which 
shows volume rendered CT data underneath the normal colon surface. It 
usually shows the translucent display in just a small portion of the normal 
image to help give context about what is seen: 
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The translucent display is generated by applying a brightly colored colormap 
with a low, constant opacity to the CT data and volume rendering from the 
same viewpoint and direction as the normal image. This same technique can 
be used to overlay PET, SPECT, CAD, shape, other modality data, or derived 
data onto the normal image. So, instead of seeing the CT data underneath the 
color surface, you would see the other data rendered below the surface, giving 
you a magic window to peer into the second modality through the first 
modality. 



3D Navigational Flight along a Pre-Computed Path with Force Feedback 

It's all too easy while flying through tortuous anatomy to lose sense of direction and lose 
one's orientation. 

The following suggests a method to enhance the user's ability to manipulate a camera in 
3D space while staying true to a pre-computed optimal path using force feedback devices 
which can apply forces in at least two directions. 

Consider any 2-axis input device; assigning the pitch of the camera to the y-axis and the 
heading to x-axis it is possible to apply a force in both x and y such that any errant path 
can be corrected. The magnitude of the force is greater the further away the camera is 
from the pre-computed path. When the camera is close to the path a gentle force will be 
applied to "guide" the user along. 




Figure 1 

In the above figure, if we have a camera with position P and direction D; we can compute 
the corrective force that must be applied to the input device to yield the direction D L 
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Perceived Endoluminal Flight Speed 

A doctor performing diagnostic examinations of colon lumen using 3D endoluminal 
flight must be able to effectively and accurately process the information that is presented 
during flight. In addition to other factors, the speed of flight determines how much and 
how well information is being presented, and as such can affect how quickly a doctor can 
process views accurately. Though traveling at a constant speed, as measured in 
millimeters/second, the speed of flight as perceived by the user is dependant on the 
distance from the viewpoint to the nearest point on the colon lumen surface. At a 
constant speed, the less insufflation that an area of the colon received, the more difficult 
it becomes for the user to focus on a particular area on the lumen wall, because of the 
perception of increased flight speed. As a user is flying thru the colon lumen, the 
perceived changes in flight speed through areas of varying insufflation can be very 
distracting to the user. Thus, it is necessary to bring the perception of flight speed during 
flight to as close to a constant as possible. 

The method used to modulate the actual flight speed in order to maintain a constant 
perceived flight speed, if the user requests such modulation, was to first determine an 
upper and lower threshold of insufflation widths (the straight-line distance from the 
lumen surface point closest to the field of view to the closest point on the opposite lumen 
wall) at which to modulate the actual flight speed. While flying through areas of 
insufflation width greater than the upper threshold, the amount of change in the perceived 
speed was minute enough such that it was not distracting to the user, and as such did not 
need modulation. For widths less than the lower threshold, the actual speed in modulated 
using the lower threshold width. A neighborhood sample of insufflation widths are taken 
and averaged, so that sharp changes in insufflation width would not result in undesirable 
large changes in flight speed. Finally, the resulting change in velocity is calculated as a 
percentage of the averaged radius to the upper threshold radius. 

The result is a gradual reduction of actual flight speed as the user's field of view 
encounters and passes thru areas of low insufflation, resulting in little to no perceivable 
increase in flight speed, which is much less distracting to the user, which aids them in 
maintaining a concentration on the lumen surface during the examination. 
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Medical Visualization as a Service 

The problem with existing hardware/software solutions for 3D/4D medical visualization 
is that it is product-oriented. This means that the typical user of medical visualization 
must purchase and install software and/or hardware to perform tasks. This requires 
considerable investment and resources and training to maintain. With the ever-increasing 
connectivity provided by the World-Wide Web (WWW), there is no longer a necessity to 
deploy software as a product. Instead, the medical visualization can be offered as service 
to be used on demand and without considerable initial investment. By having the service 
run in a server-centric environment, the investment is shifted to the service provider. 

In the proposed scenario, a user of medical visualization such as a radiologist will select a 
service provider and set up a secure account with the provider, thus establishing a secure 
link for the transferring of medical data. The user will then select a 3D or higher- 
dimensional dataset from their CT scanner or PACS archive and securely send the data to 
the service provider using the WWW. The service provider will perform a manual or 
automatic post-processing (such as segmenting organs, performing diagnostic procedures, 
or calculating features). The user will then be able to obtain the results in real-time over 
the secure network. The user will be able to interact with the server over the network to 
adjust the visualization or perform more post-processing as needed. When the user is 
satisfied with the results, the images can be automatically stored in the local PACS 
archive for long-term storage. 

An alternative is for the service provider to automatically send back the results when 
completed to the user's PACS archive without user interaction. 

An advantage of this invention is that the user is not required to purchase or maintain 
software and hardware in order to obtain the desired services. Also, upgrades to the 
processing algorithms on the server are transparent to the end user. 
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Automated organ extraction and post-processing 
Background of the Invention 

The present disclosure relates to the automatic processing and distribution of medical 
images. 

Radiologists are able to examine the inner structure of the human body by examining 2D 
cross-sectional images generated by CT or MR. Recently, the thickness of these slices is 
shrinking exponentially (by a power of two every 12-18 months) and the number of slices 
necessary to cover the same anatomy growing exponentially. This leads to the situation 
in which the quality of the diagnostic information is increasing, but the means to digest 
the growing amount of information is rapidly decreasing. Thus, new techniques are 
necessary to efficiently manage the growing amount of information. 
Three dimensional (3D) imaging is one means to rapidly assimilate many layers (or 
slices) of information simultaneously. This generally requires the use of specialized 
hardware, software, and technical personnel to specify and generate the images and to get 
them into the hands of the doctors for diagnosis. One of the most user-intensive steps is 
to separate the diagnostically useful 3D raw data from the not useful in a step that is 
usually referred to as segmentation. It requires a general knowledge of the desired 
anatomy and it usually requires several steps to identify and separate (or cull) the 
information using user-interactive techniques such as drawing, cutting, thresholding, 
filling, growing, and shrinking. These techniques are used either on a sequence of 
adjacent 2D slices or directly within a virtual 3D space on the computer screen. After the 
desired objects are separated from their background, a set of images are prepared that 
provide diagnostically useful information to the doctors. The images are usually 
prepared and stored in electronic form for review directly on a computer screen or 
sometimes printed directly to some physical medium such as film for use in a standard 
radiologist's light box. The images can be 

• select 2D images from the original acquisition 

• 2D multi-planar reformatted (MPR) images either in an axis orthogonal to the 
original image plane or in any axis, 

• curved MPR images in which all the scanlines are parallel to an arbitrary line and 
cut through a 3D curve, or 

• 3D images using any projection scheme such as perspective, orthogonal, 
maximum intensity projection (MIP), minimum intensity projection, integral 
(summation), to name a few. 

Furthermore, fused or combined images from multiple modalities (CT, MRI, PET, 
ultrasound, etc.) using any of the image types mentioned above can be generated to add 
to the diagnostic value, once the anatomy has been matched between the separate 
acquisitions. 

In addition to the type of images desired, the appropriate number, size, color, quality, 
angle, speed, direction, thickness, and field of view of the images must also be selected. 
This choice varies significantly from doctor to doctor, but it is usually related or 
proportional to the size, shape, and/or configuration of the desired object(s). 
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Because of all these reasons, 3D imaging is time consuming to generate because it 
requires knowledge of the anatomy for object selection and image generation. It is 
desirable to automate these processes to eliminate much of the costly time and tedium 
required to generate the final output. 

Summary of the Invention 

Many technologies are available that, when combined, can eliminate most or all of the 
user-interaction required to generate diagnostically useful images. There have already 
been techniques developed to automatically identify and segment the colon, lung, bones, 
heart, brain, and blood vessels from a scan of the human body. Nearly every part of the 
body can be automatically identified and segmented if sufficient distinguishing features 
are available. Sometimes a little bit of user interaction is required to direct the computer 
algorithms to focus on the correct part of the body. For instance, the user may be 
required to provide on or two clicks on the correct blood vessel since there can be tens to 
hundreds to choose from. Other algorithms may be more highly automatic and require no 
user interaction, such as the identification of the lungs, which are highly visible due to 
their distinct structure. 

At a higher level of refinement are automatic algorithms that take a segmented body part, 
such as the lungs or the colon, and attempt to identify and diagnose abnormalities in the 
part. These algorithms are generally called computer aided diagnosis (CAD) algorithms 
and are becoming increasingly more acceptable as a factor in medical diagnosis. These 
algorithms may provide further information about the body part such as principal 
dimensions, central axis, likelihood of malignancy, growth rate, potential severity, and 
details of the structural composition. All of this information is desirable to have available 
alongside or imprinted directly onto the final images. 

In order for a user to generate a set of images, they generally follow a prescribed protocol 
or set of instructions that generates a set of images with the most diagnostically useful 
information for the doctor. Because every person's anatomy is unique, the instructions 
are all relative to the individual anatomy. For example, it may be desired to generate a set 
of images through the carotid artery, a branching structure that makes visible the three 
primary vessels at the bifurcation all in a single plane. There is one unique plane that 
passes through the three vessels and an MPR image can be aligned to the plane to image 
the vessels most clearly. That MPR image plane can be slid back and forth parallel to 
itself to generate a set of images that together cover the entire vascular structure. Another 
doctor may desire a set of images that takes the same three vessels, renders them using 
3D volume rendering from the front side of the patient and rotates the object throughout 
360 degrees around a vertical axis producing 36 color images at a specified resolution, 
one image every ten degrees. Still another doctor may desire to have each of the three 
vessels rendered independently using 3D MIP projection every 20 degrees, thereby 
producing three separate sets of images (movies) each with 1 8 frames. 



11 



Physician requests body scan 

i 

Operator scans body 

z i 

Desired protocol indicated 

.1 

Computer identifies and segments 
object 

I 

Computer generates images according 
to protocol 

A 

Images transferred to electronic 
storage medium for physician access 

Figure 1: Process diagram of automated image generation 

Once the structure of the anatomy is identified and segmented, it is entirely possible for 
the generation of images to be automated by a computer, once the desired protocol is 
indicated (see Figure 1). When a body scan is requested, the purpose of the scan is always 
indicated by the physician. The desired protocol needs to be identified to the automatic 
algorithms to begin the segmentation and image generation process. 

Detailed Description 

The automatic identification and segmentation of vascular structures is of particular 
importance because vascular studies represent a large proportion of the studies done in a 
real clinical setting, partly influenced by the current trend toward cardiovascular disease 
in the population. Vascular structures are a complex structure because they consist of 
multiple, branching tubular objects that all are numerous within the body and get 
increasingly smaller with each level of branching. Fortunately, image quality and 
resolution are increasing rapidly to the point where the majority of important vessels are 
highly visible if the scan is properly performed. For the computer to identify a particular 
vascular structure, it must have some knowledge of how the structure should appear 
within the raw data. In particular, a vascular structure should contain many of the 
following characteristics: 

• Location within the body 

• Direction relative to the body 
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• Length of section of vessel 

• Curvature of the vessel centerline 

• Axis of curvature 

• Diameter 

• Area 

• Intensity of interior of vessel 

• Texture features of interior of vessel (e.g., grainy, homogenous) 

• Intensity of exterior of vessel 

• Branch location and connectivity to other vessels 

The structural characteristics of a vessel need to be defined parametrically along the 
length of the vessel which, in turn, defines the overall shape of the vessel. Furthermore, 
the characteristics of the vessel all need to defined as not only a target value, but also 
with a desired range of values for each. For instance, the diameter needs to be defined not 
only as a target diameter as well as a minimum and maximum diameter. The minimum 
and maximum values can be interpreted as both hard limits (everything outside of the 
range is rejected) or as highly desirable such that a majority (fixed percentage) of the 
vessels will fall within the given range, similar to a standard deviation. Furthermore, the 
characteristics can apply over various ranges, small or large. For instance, the local 
direction of a vessel may vary significantly, but the overall, composite direction may be 
relatively invariant. Also, the relative weight of the characteristics may be varied in order 
to better select the correct vessel when faced with a choice among several options. 

The vessels can be identified as follows. First, the probability of the presence of a vessel 
is determined at every voxel location in an input dataset of CT, MR, or other modality 3D 
image. The technique used here varies, but can be any technique that recognizes the 
shape and/or intensity profile characteristic of a vessel. Then, a higher-level recognition 
algorithm connects the voxels within the dataset that meet a certain threshold. One 
method to do this is by finding seed points (of the highest probability of being a vessel) 
either randomly or deterministically. These seed points can be grown in the most 
probable vessel direction using the vessel probability of the neighboring voxels. The 
seeds then grow into long sets of voxels that are probably the locations of vessels. These 
sets of connected regions are then analyzed to determine the most likely groups that 
represent a vascular structure. The structure will probably consist of multiple branching 
vessels and these branches must be noted and identified as characteristics. Then, each of 
the probable vessels is analyzed to extract as much of the characteristics over the length 
as possible. At any point after recognition, the vessel centerline can be extracted and 
optimized (e.g., using the techniques disclosed in provisional application serial number 
60/525,603 which is incorporated herein by reference) for better visualization. The next 
phase is the matching of the probable vessels in the unknown dataset with the known 
anatomical vessels. This may be done by any matching method that matches multiple sets 
of features to multiple sets of features. One example method for matching is that 
described in U.S. application serial number 10/496,430, incorporated herein by reference. 
The general idea is to get the best-fit match between the two sets first, then to eliminate 
the two sets from consideration and repeat until all unknown vessels are determined. By 
finding the most probable match first, it helps to find the other vessels due to the 
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connections and placement of the other vessels in relation to the matched vessels. In 
general, the method to recognize and identify vascular and other structures in the body is 
a multi-level approach which recognizes small, local features as well as larger, aggregate 
features based on groups of smaller features. The number of levels as well as the 
direction of recognition (bottom-up or top-down) is not important. 




Note that the method can be extended to recognize other objects as well as vessels. For 
instance, it can be used to identify bones in the body such as vertebrae or other organs. 
To do so, the underlying mechanism for identifying prospective objects needs to be 
extended to locate non-tubular shapes. The advantage is that the more objects are 
recognized and placed within the body; the more can be inferred about the remaining 
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parts. Tubular shapes are just one example of a recognizable feature. They are identified 
at the lowest level by their characteristic signature - a cylindrical shape generally 
composed of a higher intensity cylinder embedded in a lower intensity surrounding 
(although the intensities could just as easily be reversed). A mathematical 
characterization, such as a Hessian filter, a Hough transform, or any other 
shape/structure/texture recognition or combination of these can be employed to 
characterize individual features, and then these features can be hierarchically determined 
using characteristic groups of features arranged in a characteristic way. These groups, in 
turn, can be categorized into features and again grouped to form even higher level 
features. The net effect is a hierarchical recognition of objects composed of multiple 
distinctive features. 

After the vessels or other body parts are recognized by the automatic algorithms, they can 
be labeled for future manipulation and visualization. If desired, various automatic 
visualization techniques can be applied in order to aid in diagnosis. Often, these 
visualization techniques are applied interactively by a technician, as described above, but 
the computer can automatically generate the desired visualization to save time and effort. 

The desired segmentation and visualization protocols can be automatically determined 
based on some information either within the image data (if it looks like a heart, then 
process it as using the heart protocol; if it looks like a lung, processes it using the lung 
protocol), meta-data attached to the image (e.g., using one of the 'tag' fields in DICOM), 
or by user-input configuration at the computer console. 

In more detail, one possible mechanism by which to indicate the desired protocol is for 
the scanner operator to input the protocol to the scanner which encodes this information 
along with the image data after the scan is completed. Another mechanism is for a person 
to select on the computer the desired protocol from a list of available protocols. Another 
mechanism is for the computer to automatically determine the protocol using whatever 
information is available in the image data (if it looks like a heart, use the heart protocol, 
etc.) and the metadata that comes along with each image (e.g., the referring physician's' 
name is "Jones" and he prefers protocol "A" for heart scans, except for short, female 
patients with heart scans he prefers protocol "B"). As can be seen, the possibilities for 
automatic selection are virtually unlimited because the protocol can be derived from so 
many factors including the unique data scanned in every image. 

For every type of anatomy, there is usually as set of visualization techniques that are 
optimal for diagnosis. For looking at vessels, it is desirable to visualize the vessel in a 
curved MPR view and a rotating 3D view. For lung nodules, a "cartwheel" projection that 
shows a small square oblique MPR view that rotates 1 80 degrees around a central axis of 
a suspected lung tumor. For virtual colonoscopy, it is desirable to have a 3D flythrough 
along the entire length of the colon. To obtain these images, a technician must spend time 
creating these views or the doctor must create these views himself. Once the views are 
created, a subset of the views is sent to a digital archive for short and/or long term 
storage. From there, the physician can view the original source images along with any of 
the derived images to make a diagnosis. The doctor will either locate suspicious 
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locations in the original images and confirm diagnosis using the derived images or vice 
versa. In any case, the generation of the protocol specific images can be automatically 
generated and saved in a convenient location using the automatic methods described 
herein. 

The general practice in modern radiology departments is for all digital images to be 
stored in a Picture Archiving and Communication System (PACS). Such a system 
centralizes and administrates the storage and retrieval of digital images from all parts of 
the hospital to every authorized user in a hospital network. It usually is a combination of 
short-term and long-term storage mechanisms that aim to provide reliability, redundancy, 
and efficiency. To read images within such a system, the radiologists usually selects a 
patient and the "series" or sets of images in the study are recalled from the PACS and 
made available for examination on a PACS viewer usually on a standard personal 
computer. 

The following is a description of the creation of customized movies of vessels. After 
carotid vessels are automatically found and segmented, a set of customized movies are 
created. These movies are defined in an XML script file that is customizable on a user- 
by-user basis. An animation script consists of a set of animations as shown below in the 
following sample. 

<?xml version="l,0" encoding="utf-8"?> 
<animations version="l . 0 M > 

<animation name^Tull Volume" version="l . 0"> 

<preset version="l . 0" name="CTA, Carotid" number="0" /> 

<componentVisibilities version="l . 0" defaultvisible="true"> 

< /component Visibilities> 

<animator version="l . 0"> 

<action version="l . 0" typename=" Viatronix. v3D. Animations. AnimationActionRotate3D, 
Viatronix. v3D. Visualization'^ 

<AxisOfRotation>y</AxisOfRotation> 

<DegreesPerImage>10</DegreesPerImage> 
</action> 

<capture version="l . 0" typename="Viatronix . v3D. Animations .AnimationCaptureMovie vi 
Viatronix. v3D. visualization'^ 

<ImageWidth>256</ImageWidth> 
<ImageHeight>256</ImageHeight> 

<FramesPerSecond>10</FramesPerSecond> 
</capture> 
<renderers> 

<renderer version="l . 0" typename="Viatronix. v3D. Visualization. RendererCpu3D, 
Viatronix.v3D. Visualization" /> 

<renderer version="l . 0" typename="Viatronix .v3D. Visualization . RendererOverlav3D, 
Viatronix.v3D. Visualization" /> 

<renderer version^" 1.0" typename=" Viatronix . v3D. Visualization . RendererOverlay, 
Viatronix.v3D. Visualization" /> 
</renderers> 

<renderingQuality value="def ault"/> 
<font family= w Arial n size="12"/> 
</animator> 
</animation> 

<animation name="All Vessels" version="l .0"> 

<preset version="l . 0" name="CTA f Carotid" number=="2" /> 
<componentVisibilities version="l . 0" defaultVisible= ,, true f, > 
<componentVisibility narae="Background" visible= M false" /> 
</componentVisibilities> 
<animator version="l . 0"> 

<action version-"1.0" typename=" Viatronix .v3D. Animations . AnimationActionRotateSD, 
Viatronix. v3D.Visualization"> 

<AxisOfRotation>y</AxisOfRotation> 
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<DegreesPerImage>20</DegreesPerlmage> 
</action> 

<capture version^. 0" typenarue="Viatronix. v3D. Animations . AnimationCaptureMovie vi, 
Viatronix. v3D. Visualization'^ - 

<ImageWidth>512</lmageWidth> 

<ImageHeight>512</ImageHeight> 

<FramesPerSecond>15</FramesPerSecond> 
</capture> 

<renderers> 

tenderer version="l . 0" typename="Viatronix. v3D. Visualization. RendererCpu3D, 
Viatronix. v3D. Visualization" /> F ' 

tenderer versions . 0" typename=»Viatronix. v3D. Visualization. RendererOverlay3D, 
Viatronix. v3D. Visualization" /> y ' 

<renderer version="l . 0" typename=="Viatronix. v3D. Visualization. RendererOverlay, 
Viatronix. v3D. Visualization" /> y ' 
</renderers> 

<renderingQuality value="highest"/> 
<font fairtily="Arial" size="12"/> 
</animator> 
</animation> 
</animations> 

In this sample, the first animation is a 256 2 image of the entire dataset in 3D rotating 
about the vertical (Y) axis with 36 frames at 10 frames per second. It chooses the first 
(index 0) preset that matches the "Carotid, CTA" name. It turns on all the components 
and renders with the default rendering quality. The second animation is eighteen 5 12 2 
images rendered at the highest quality at 15 frames per second. It turns on all the 
components, and then specifically turns off the Background component, leaving just the 
automatically segmented vessels. 

If we had specified regex="Internal" instead of name="Background" within the 

component visibility, it would have turned on all of the internal carotid arteries because 
the regular expression "internal" matches both "Left Internal Artery" and "Right 
Internal Artery." We could have also used regex="Lef t" in order to display all the left 
side arteries or regex="Right " for just the right side arteries. The use of name= implies 
that only an exact match should be considered while the use of regex= implies that any 
matching of the regular expression in a component includes it for consideration. In the 
case of a tie or conflicting rules, the order of specification should override the conflict. 

The computer-generated images could also be a combination of multiple kinds of images 
and information, drawn together to provide a richer set of information in each single 
image. These images are combined and all updated synchronously to provide contextual 
or numerical information about the current diagnostic image. For instance, a vessel 
examination can benefit from having a combination of overview, vessel view, and 
detailed vessel view all combined into a single image as shown in Figure 3. 
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Figure 3: Specialized vessel view showing overview (upper left), vessel view (right), and detail view 
(lower left) 

As another example, Figure 4 shows a frame from a virtual colonoscopy animation in 
which a virtual endoscope travels along the centerline of a colon. An overview image 
showing the entire colon (yellow shape), the colon centerline (green curve), points of 
potential interest (red circles), and the current camera position (blue arrow) is overlaid of 
inset into the main image to provide additional contextual information. 



18 



Figure 4: Specialized display for virtual colonoscopy showing large endoscopic image with inset 
overview image. 

For lung studies, it is desirable to have an automatic algorithm process a data set to locate 
suspicious regions and evaluate them for malignancy or other characteristics. Then, 
various views of the suspect anatomy can be generated according to some user-specified 
protocol and images generated with supporting information possibly superimposed or 
attached as meta-data. Finally, the computer-generated results can be stored as digital 
images in a standard digital format (e.g., DICOM) in a standard digital archive (e.g., a 
PACS) so that during diagnosis of the original images, the associated computer-generated 
images can be reviewed within the conventional diagnostic environment. 

For cardiac studies, it is desirable to visualize the heart chambers using so-called long 
and short axis views. To do so, the heart must first be recognized along with its major 
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axis. Then, double oblique MPR images cut through the heart in the planes of equal 
longitude and latitude - these are the short and long axis views. With sufficient contrast 
enhancement, the blood-filled chambers (ventricles) of the heart can be segmented from 
the rest and the various computations made comparing the volume of the ventricles at 
different times during the heart cycle. The result of these computations and possibly 
textual and visual information on the results can be formatted into one or more 
information-rich images. These images can then be sent to the PACS archive or to a web 
server or some other accessible digital storage devices that is normally used as a source 
from which to draw images for diagnosis. 

Similar procedures exist for other specialized examinations such as brain, carotid, colon, 
aorta, trauma, spinal column, kidney, liver, knee, bladder, etc. The desired processing 
protocol and visualization protocol specifying the desired images can be provided and 
customized by the user. 

Additional information can be supplied and attached to the images. The information can 
either be displayed directly within the images (such as the distance from the rectum in a 
virtual colonoscopy, or the length/diameter/volume of a vessel in vessel analysis) and/or 
it can be embedded or attached to the image as meta-data. For instance, in vessel 
analysis, a full 3D description of the shape can be converted to a standard (e.g., XML) 
format an embedded into the image. If the image format is DICOM format, then special 
DICOM fields and extensions of them (i.e., private fields) are available to store additional 
data. DICOM data can also hold additional binary-formatted data at the end of the image 
that is retained throughout transfer, is readable by any reader, but not visible to the 
typical user since it is beyond the visible bounds of the pixel data. If the image format is 
PNG, then additional data chunks are available within PNG to store any additional data. 

The advantage of the present invention is the automatic generation of computer- 
processed results performed according to a customizable protocol with the results being 
easily accessible within the existing diagnostic environment. 

Claims 

What is claimed is: 

1 . An automatic method for accepting a set of at least one images of at least one 
object, the method comprising: 

processing said at least one images to extract at least one known object; 
accepting a set of at least one directives specifying the desired output; 
generating at least one images in a standard digital imaging format according 
to the at least one directives; and 

transferring at least one generated images to a digital archive. 

2. A method as defined in Claim 1 wherein the digital archive is a hard disk. 

3. A method as defined in Claim 1 wherein the digital archive is a standard medical 
archive system (e.g., a PACS archive). 

4. A method as defined in Claim 1 wherein the object is an organ within a body. 

5. A method as defined in Claim 4 wherein the organ is a blood vessel. 
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6. A method as defined in Claim 5 wherein the processing of at least one images 
extracts tubular objects by filtering the input data with a filter for tubular objects. 

7. A method as defined in Claim 6 wherein the tubular objects are identified by 
matching with a set of known blood vessels for characteristics such as position, 
direction, curvature, diameter, connection to other objects, and relation to other 
objects. 

8. A method as defined in Claim 1 5 wherein the at least one generated images 
include a 2D multi-planar reformatted image displaying cross-sectional cuts 
through the object. 

9. A method as defined in Claim 1 , wherein the at least one generated images 
include a 3D view of the at least one object. 

10. A method as defined in Claim 1 , wherein the at least one directives include the 
generation of a plurality of images that when played in sequence simulate a 
continuous transformation (e.g., rotation, translation, a combination of rotation 
and translation) of the object or viewpoint. 

1 1. A method as defined in Claim 9, wherein the plurality of 3D images simulates the 
movement of an endoscope through a visually hollow organ within a body. 

12. A method as defined in Claim 11, wherein the organ is at least one blood vessel. 

13. A method as defined in Claim 11, wherein the organ is a colon. 

14. A method as defined in Claim 11, wherein the organ is a heart. 

15. A method as defined in Claim 11, wherein the organ is a lung or part of a lung. 

16. A method as defined in Claim 1, wherein the generated images are combined 
together by juxtaposition or inset and the information synchronized if possible. 

17. A method as defined in Claim 1, wherein computer-generated meta-information is 
combined with the image and either displayed within the image or attached to the 
image as supplemental information allowing readers of the image to retrieve the 
information and obtain diagnostic numerical information, computer-enhanced 
data about the object in the image, or shape information about the object in the 
image. 
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System and Method for Building the Library of Digital Tissue 
and its Application to Lesion Detection and Staging 



Background 

The repaid advance in medical imaging technology provides much better tissue contrast 
than before. The improved tissue contrast allows detecting the subtle difference between 
normal and abnormal, or benign and malignant tissues in the medical images. More 
important, the better quality images provide much stable characteristics for digitally 
comparison of virtual samples that are taken out from image series which are acquired at 
different time sections. This made the virtual histology (or digital histology) possible. It 
also opens great opportunities for lesion or tumor staging based on medical images. In the 
following, we describe the general scheme of virtual histology first. Then, we shall 
describe an application to virtual colonoscopy as an example. 
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Figure 1. The flow chart demonstrates the work flow of virtual histology. The digital tissue library is a 
collection of typical digital samples and their intrinsic characteristics in digital world. 



General Scheme 

Figure 1 is a flow chart for the virtual histology technology. 
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Patient will follow necessary preparation procedure in order to enhance or highlight 
certain type of tissue or lesion in the images. For example, the IV contrast agent injection 
for vessel enhancement in the CT angiograph application. The preparation may be done 
at patient's home or at the scanning suite. For example, patient orally intakes barium for 
highlighting residues in the colon. In general, the patient preparation may be any kind and 
may or may not be necessary. 

A series of medical images is acquired from a subject at scanning suite after patient 
preparation. The images can be any modality with high resolution and good tissue 
contrast. The subject can be human being or animal. The computer system receives the 
medical images and post-processes them. The computer system can be directly connected 
to the image acquisition equipment or via a network. The post-processing can be multiple 
purposes. For example, image enhancement, noise deduction, organ segmentation, initial 
detection of abnormalities, building of 3D model for display, and etc. 

After post-processing, the images will be loaded and display in medical imaging 
workstation in various display modes for physician review. The initial detected results by 
computer algorithm at post-processing step will be labeled and provided to physician for 
diagnosis assistance. After physician confirming an abnormality, he/she can use the 
mouse click on the target. Then, the system will automatically/interactively extract out 
the target sub-volume encapsulate that abnormality region. The sub-volume is the so- 
called Digital Sample for the abnormality. The sub-volume is not a merely group of 
voxels. It is extracted based on the minimum size for representing a certain lesion or 
abnormality tissue functions. It should provide the basic functional clue for pathology 
analysis. 

A database of digital samples will be built. The initial digital samples in the database will 
be used for feature selection. The unique features related to a specific type of abnormality 
will be extracted for all digital samples of that type. The features must be essential 
characteristics for the specific type of abnormality. In other words, an indicator of tissue 
type for that kind of abnormality can be constructed based on those features and the 
indicator must have high sensitivity for characterization of the specific abnormality. 

Note, the features and the built indicator for a specific tissue type are associated to the 
digital sample, as a whole tissue sample with certain bio-function rather than a group of 
voxels. This is completely different from that of conventional CAD (computer aided 
detection) approaches. In conventional CAD approach, the extracted feature is related to 
independent voxel or a group of voxel, where the entire digital sample had never been 
considered at its feature extraction stage. In other words, the conventional CAD approach 
works on collection of fragment information of a tissue type and try to put them together 
to get a conclusion. Instead, the virtual histology works on the complete tissue sample as 
whole at the very begin. The features that are extracted from digital sample must be 
global rather than voxel-wise to the tissue type or type of lesion. 
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For certaion lesion type, the initial features and the tissue indicator will be collected and 
developed in a digital sample library. More precisely, the digital sample library is a 
categorized database for features and digital sample indicators. When new digital sample 
is gotten, the features that extracted from the new sample will compare to those in the 
library. Using the tissue indicator of the library, one can get a conclusion that the new 
digital sample is most probably a certain type of known tissue in the library. 

Data-mining technology should be employed for improving and enriching the library 
when more and more digital sample are available. The digital sample can stratified in 
different categories based on type of lesion or different stage of the same type of lesion. 
For example, benign and malignant polyps. 

The most important point here is the consistency of the digital sample in physics 
characteristics. In other words, we assume that the medical images quality guarantee the 
same tissue type will have similar properties regardless of subjects and acquisition day. 
This is the basic assumption for the feasibility of virtual histology. Besides the image 
quality, the method of extracting digital sample is essential. It must segment out the 
correct sub-volume in a consistent way regarding the size, contour, voxel resolution, and 
the normalized voxel intensity. 

Virtual Colonoscopy as an Example 

The image acquisition procedure of virtual colonoscopy can be the routine one. The post- 
processing and display modes for physician review can be any of available. The only 
thing that triggers the virtual histology is a mouse click. By clicking on the suspicious 
polyp region, the virtual polypectomy algorithm is applied. The selected sub-volume of 
the target polyp will be delineated as the digital sample. 

The initial suspicious polyp location can be either provided by CAD algorithm or by 
radiologist manual input. 

In order to facilitate better understand, we use the shape feature as example to develop a 
polyp indicator. That should not limit our invention using only shape features for polyp 
indicator. 

Polyp is growing inward to the lumen. Its shape is different from those of haustral fold 
and normal colon wall surface. It roughly has a convex cap like top with/without a stake. 
By developing a local intrinsic landmark system on the polyp sub-volume, a shape 
template can be developed, which should be invariant to translation and rotation (ref. 
Patent application by D. Chen et al. (2003): Computer-Assisted Detection of Lesion 
in Volumetric Medical Image), which is incorporated herein by reference. The shape 
templates that are collected from a training set can be classified to represent polyp (of 
different types), haustral fold, and normal colonic surface. A library of shape templates 
will be developed based on available digital sample of polyp. When new case comes in, 
the newly collected digital sample will be compared with the templates in the library for 
tissue confirmation. Figure 2 shows a polyp in endoluminal view. Figure 3 shows the 
extracted digital polyp sample that is coded in different color in the endoluminal view. 
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The maximum and minimum diameter and its volume are displayed. Figure 4 shows a 
digital sample of the dissected polyp that is stored in the library. 
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Figure 3. The polyp digital sample is coded in different color in the endoluminal. The 
maximum and minimum diameters and volume of the polyp are displayed. 



26 




Figure 4. The dissected polyp digital sample is showed in 3D view. 
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